Reduction Optimization in Heterogeneous Cluster Environments

Network of workstation (NOW) is a cost-e ective alternative to massively parallel supercomputers. As commercially available o -the-shelf processors become cheaper and faster, it is now possible to build a cluster that provides high computing power within a limited budget. However, a cluster may consist of di erent types of processors and this heterogeneity complicates the design of e cient collective communication protocols. For example, it is a very hard combinatorial problem to nd an optimal reduction schedule for such heterogeneous clusters. Nevertheless, we show that a simple technique called slowest-noderst (SNF) is very e ective in designing e cient reduction protocols for heterogeneous clusters. First, we show that SNF is actually an approximation algorithm with competitive ratio two. In addition, we show that SNF does give the optimal reduction time when the cluster consists of two types of processors, and the ratio of communication speed between them is at least two.